Repository navigation
fix(notebooks): fine-tune notebook fits temperatures outside runtime clamp, so served calibration differs (#637) - #642
Conversation
|
Holding: Please split this PR to keep the notebook temperature clamp and test only, and drop the external php links and unicode-escape churn. |
|
Thanks for the feedback. I’ve updated PR #642 as requested. The PR is now limited to the notebook temperature clamp using TEMP_MIN/TEMP_MAX and the corresponding regression test. I also removed the unrelated PHP/Laravel documentation changes and Unicode/serialization churn. |
Bruce-Yii
left a comment
There was a problem hiding this comment.
This is now exactly the split you asked for, and the remaining change is right. I checked it against current main rather than taking the description on faith.
The bug is real, and the mismatch is worse than "differs"
notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb clamped the fitted temperature to (0.1, 10.0), while the runtime it has to agree with bounds the same quantity differently:
laya/common.py:552 TEMP_MIN = 0.5
laya/common.py:553 TEMP_MAX = 5.0
laya/common.py:556 def clamp_temperature(t, lo=TEMP_MIN, hi=TEMP_MAX)
So a temperature the notebook declared fitted and persisted was re-clamped on the way out — the notebook's own fit was the thing being silently discarded, not just a rounding difference.
RED → GREEN on current main
I took the PR's two files onto 6d942c9 and ran only that suite, so both arms are the same tree with one file swapped.
| tree | result |
|---|---|
main notebook + this PR's test |
1 failed, 5 deselected — AssertionError: 10.0 not less than or equal to 5.0 |
| this PR's notebook + its test | 6 passed, 45 subtests passed |
The failure is the assertion the fix exists to support, on the high side, with the real fitted value — not an import or a fixture error.
It also merges clean, which is the part that would otherwise cost a round trip
The PR's base is 9d95567 (v0.3.21), so the diff view shows 226 files. That is base age, not scope — the authored change is the two files and +23/-4:
notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb | 6 +++---
tests/test_calibration_persistence.py | 21 +++++++++++++++++++-
Applying just those two files to 6d942c9 merges with no conflict.
One non-blocking note
The new check pins only the high bound. I looked for an input family that drives the fit below TEMP_MIN = 0.5 and could not construct one — across a sweep of logit and target ranges the lowest value reachable under main's (0.1, 10.0) clamp was 0.519. So the low side may well be unreachable through this code path, which is why I am not asking for anything here. Flagging it only so the asymmetry is a decision rather than an oversight if that turns out to change.
Approving.
|
Merged for 0.3.23. Verified on the integrated branch, not on the PR in isolation: every suite in the Thank you. |
The notebook rewrite removes the `%%writefile` cell this guard scraped for the config export and for NandhaKishorM#642's temperature clamp. Point both at laya.finetune.train_rlcd / fit_temperature, which own that code now, so the guard still runs instead of erroring on an empty cell list.
Summary
Fixes #637.
This PR fixes a discrepancy between the temperature produced during post-training calibration and the temperature used by the production
laya.Agent.In
notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb,fit_one_temp()previously clamped fitted temperatures using:However, when a checkpoint is loaded in production,
laya.Agent(laya/agent.py) passes the stored temperature throughclamp_temperature()(laya/common.py), which enforces the serving bounds:This meant that the notebook and production environment could use different temperature values.
The Discrepancy
If
fit_one_temp()produced a temperature outside the serving range—for example,9.886or7.10—that value could be persisted torl_agent_config.jsonunchanged.When the checkpoint was subsequently loaded,
laya.Agentwould clamp the value to the production range. For example:As a result, the calibration performed during fine-tuning did not necessarily reflect the behavior of the model after deployment:
This could also result in a warning when loading checkpoints containing temperatures outside the serving bounds.
Changes
1. Kaggle Fine-Tuning Notebook
Updated
notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynbto use the canonical temperature bounds fromlaya.common.TEMP_MINandTEMP_MAX.fit_one_temp()to clamp fitted temperatures using:This ensures that the temperature produced during calibration is already within the same range enforced during serving.
2. Regression Test
Added
test_notebook_fit_one_temp_clamps_to_common_boundstotests/test_calibration_persistence.py.The test extracts
fit_one_temp()from the notebook and verifies that its output remains within[TEMP_MIN, TEMP_MAX], including when presented with extreme logit inputs.This prevents future changes from reintroducing a mismatch between calibration and serving behavior.
Verification
Ran syntax and compilation checks successfully: